Goto

Collaborating Authors

 careful data engineering transforming data


Great Machine Learning Needs Careful Data Engineering Transforming Data with Intelligence

#artificialintelligence

A new TDWI Checklist Report examines best practices for data engineering and management to support machine learning with a focus on collecting, cleansing, transforming, and governing new and big data for analysis. In a new TDWI Checklist Report, "Five Data Engineering Requirements for Enabling Machine Learning," Fern Halper, vice president and senior director of TDWI Research for advanced analytics, notes how a new generation of data is reinvigorating interest in AI and machine learning -- and providing new challenges to enterprises of all sizes. Machine learning does what its name implies -- it is a system that learns to identify patterns by examining data. There are two approaches: supervised (where the system is given the desired target and learns to predict the same outcome based on attributes) and unsupervised (where there are no predefined outcomes, and once trained, the model is tested against additional data to make sure the model is valid). Although still in the early mainstream phase of adoption, machine learning is being deployed in a wide range of use cases, including recommendation engines, fraud detection, churn analysis, and cybersecurity.